Simultaneous Model Selection and Optimization through Parameter-free Stochastic Learning

Stochastic gradient descent algorithms for training linear and kernel predictors are gaining more andmore importance, thanks to their scalability. While various methods have been proposed to speed up theirconvergence, the model selection phase is often ignored. In fact, in theoretical works most of the timeassumptions are made, for example, on the prior knowledge of the norm of the optimal solution, while inthe practical world validation methods remain the only viable approach. In this paper, we propose a newkernel-based stochastic gradient descent algorithm that performs model selection while training, with noparameters to tune, nor any form of cross-validation. The algorithm builds on recent advancement inonline learning theory for unconstrained settings, to estimate over time the right regularization in a data-dependent way. Optimal rates of convergence are proved under standard smoothness assumptions on thetarget function, using the range space of the fractional integral operator associated with the kernel.